fix(ingestion): prune stale vectors when a changed file is guard-rejected - #237

Detail bug report: View on Detail

Closes #224

Bug

When a previously-ingested local docs file's on-disk content changed to a body HtmlContentGuard rejects (e.g. a re-fetched mirror file overwritten by a Cloudflare "Just a moment…" challenge or a 404 Not Found body), the content-guard rejection branch in LocalDocsFileIngestionProcessor.prepare() returned a failedFile after only copying the file to quarantine. It discarded the already-computed requiresFullReindex and markerContext, so it never deleted the URL's prior Qdrant points, never pruned the now-orphaned local parsed-chunk state, and never advanced the file marker.

Consequences:

• Stale prior-good vectors stayed indexed and citable under the URL while the file on disk (and the upstream URL) served the rejected body — a retrieval returned a citation whose snippet (stored doc_content) disagreed with its url.
• The marker kept the old fingerprint, so the mismatch re-fired on every subsequent run while the URL stayed rejected.
• This was a regression: the sibling processExcludedPage path already honored the correct contract for "previously-indexed URL no longer indexable" (delete vectors, prune local state, re-mark with empty chunk hashes), but the guard branch skipped all three steps.

Fix

Route the guard-rejection branch through the same cleanup sequence processExcludedPage uses, while preserving the failedFile outcome so batch-stop semantics are unchanged:

• prepare() captures requiresFullReindex and markerContext and passes them into the deferred quarantine transition (previously only file and rejectionReason were closed over).
• quarantineRejectedFile runs the forensic quarantine copy as best-effort (its try/catch is narrowed to the copy only), and when requiresFullReindex it deletes the URL's vectors (deleteByUrl), prunes obsolete local state (pruneObsoleteLocalStateAfterReplacement(..., List.of())), and writes a fresh FileIngestionRecord with the rejected file's new fingerprint and chunkHashes = List.of(). The phase stays content-guard (or quarantine-write when the copy fails), so processBatch still stops on rejection.
• The fix mirrors the in-codebase rule for the same precondition, keeping index integrity independent of the forensic-copy write. Phase/details are tracked as String locals to stay under the PMD CouplingBetweenObjects threshold (the class sits at 35; an IngestionLocalFailure local would have pushed it over).

Testing

• New regression tests in LocalDocsFileIngestionProcessorTest (Mockito, no external services):
  • shouldNotLeaveStaleVectorsWhenPreviouslyIngestedFileIsGuardRejected asserts deleteByUrl, pruneObsoleteLocalStateAfterReplacement, and markFileIngested(..., List.of()) are all invoked when a previously-ingested file is guard-rejected.
  • shouldNotLeaveStaleVectorsWhenQuarantineWriteFailsForPreviouslyIngestedFile stubs the quarantine write to throw IOException and asserts the same cleanup still runs, pinning the best-effort ordering.
  • shouldBeIdempotentOnNextRunAfterGuardRejectedPreviouslyIngestedFile drives two consecutive runs: the first rejects and cleans up; the second, over the same rejected bytes, returns a terminal skippedFile with no re-quarantine, no re-cleanup, and no chunking — pinning that the marker advance makes the next run idempotent.
  • All three fail on the pre-fix code (deleteByUrl never invoked) and pass with the fix.
• Routine checks (all green): compileJava/compileTestJava; Spotless (Palantir) check; PMD pmdMain/pmdTest (no CouplingBetweenObjects violation); SpotBugs spotbugsMain/spotbugsTest; make lint (ast-grep rules, chat-model SSOT check, frontend oxlint/eslint/svelte-check); and the CI lane (./gradlew build -x test + spotbugsMain pmdMain).
• Regression scope: the whole LocalDocsFileIngestionProcessorTest class (31 cases), the full com.williamcallahan.javachat.service.ingestion.* package (101 cases), and the full Java unit suite (1165 cases) pass with zero failures — the existing first-time-rejection tests (shouldStopBeforeLaterFileWhenContentIsRejected, shouldStopBeforeLaterFileWhenQuarantineWriteFails), the sibling processExcludedPage contract, and the chunk-replacement / marker / prune ordering are all unchanged.
• Qdrant contract (verified against live Qdrant 1.18.3): started the project's Qdrant container (make compose-up), confirmed /healthz passed, and ran the repository's synthetic hybrid-contract check (make test-qdrant-integration), which exercises the Qdrant REST surface (schema, indexes, upsert, exact count, hybrid filter, discovery, citation scroll) that the fix's deleteByUrl operation targets. Passed.
• End-to-end (not verified): two live procedures could not be run. (1) Overwrite a mirror file with a guard-rejectable body, re-ingest, and confirm Qdrant holds zero points for the URL and retrieval returns no citation for it; and (2) overwrite the file again with good content and confirm replaceUrlDocuments recovers the URL. Both require a reachable embedding gateway. The configured gateway (api.llm-gateway.iocloudhost.net) is reachable but rejects the only available placeholder credential with HTTP 401 Invalid API Key, and no real OPENAI_API_KEY is provisioned in this environment, so the live embedding round-trip could not run. The unit tests above pin the cleanup behavior these procedures would exercise end-to-end.

Automatic Fixes PRs can be configured here.